Information has been noted for hand by pen on paper since several hundreds of years. However, paperbased information has drawbacks, such as being permanent and difficult to change and distribute.
Recently, in personal computers, information may be entered by keybords and displayed by display devices, like CRT or LCD screens. However, computerbased information has drawbacks, such as being difficult to enter by keyboards and difficult to read on a computer screen.
Several suggestions exists for digitizing handwritten information for transferring such handwritten information to digital devices, such as computers.
As soon as the information has been entered in the computer, it may be communicated to others, such as by e-mail or SMS or via an Internet connection or as a fax via a fax modem.
In a computer context, a mouse is often used for selecting information and acting upon such selected information. A typical situation involves draging a mouse pointer over characters of information displayed on the screen to select said information. Then, the mouse pointer is pointed at a certain position on the screen, such as in a menue system or pallette, in order to initiate certain actions to be performed upon the selected text, such as changing the font type or size.
The same type of selection may be performed in the pen and paper situation, such as encircling a piece of information or yellow-mark information. However, since the information is fixed, no action can normally be taken upon the selected information, which however, may be highlighted to the reader.
Patent Application No. PCT/SE00/01895, filed by applicant, the technical disclosure of which is incorporated herein by reference, discloses a system in which a handheld device and a paper is used for jotting down information in a traditional way, whereby at the same time is generated a digital graph comprising one or several traces of lines of the movement of the pen over the paper, which graph may be transferred to a computer. In such a system, the advantages of using a pen and paper, which many users prefer, is combined with the flexibility of the computer to communicate and store the information to obtain the best of the two worlds. In this patent application, the paper is provided with an absolute position coding pattern, comprising dots or markings. The handheld device includes an optical sensor which registers the pattern. A mathematical algorithm calculates the absolute position in the form of coordinates for the position of the handheld device by means of the absolute positon coding pattern.
The traditional pen is in this way tranformed into an input means for the computer and the computer may be used for storing the information instead of or in combination with arranging the paper in a binder. Moreover, the information may be communicated by means of the computer. However, the information inserted by this input method is graphical and not in character coded format, which the computer is more apt to.
The information included in the graphical input comprises information which may be used for different purposes:
1) The information comprises a picture, like drawings or lines with a mutual relationship, which may be interpreted by the human being, such as characters, symbols, or a drawing. This is the very information which is put down on the paper and which the user wants to be treated in one or the other way, such as being archived or being sent to a recepient. This information is below called message information and is stored in a graphical format such as a vector format or as a collection of pixels.
2) The portion of the message information which comprises handwritten characters, can be exposed to further treatment, such as character recognition (OCR, optical character recognition or ICR, intelligent character recognition) for converting the information to a character format, such as ASCII, which may be used by the computer, e.g. for searching purposes or for cataloging. Also symbols, such as shorthand symbols or icons, may be recognized which by the user has been predefined to have a certain meaning. This information is below called character information.
3) The information may further comprise an indentification of the handheld device which is used for writing the information. The identification of the handheld device is below called pen-ID.
4) Furthermore, there is information on where at the surface the information was jotted down, so called absolute position information.
5) There may be further information, such as time information obtained from an internal clock, pressure information from a pressure sensor about the pressure of the handheld device towards the paper, and angle information about the angle between the handheld device and the paper. From the time information, speed and acceleration of the handheld device may be calculated.
The present invention is mainly focused on the information of type 4) and 5) above, such as absolute position information.
There are further known systems for obtaining absolute position information in a pen and paper system, see for example U.S. Pat. No. 5,882,434 and U.S. Pat. No. 5,051,736. Such previously known systems describe use of such information for digitizing message information, i.e. group 1) indicated above.
The information may be used in different places in the system so far described. The handheld device may comprise a picture sensor and a processor with associated memory and an energy source such as a battery. Moreover, there may be a communication device, such as a wire or a connection to a computer or an IR link or a shortrange radio connection. There is also a pressure sensor which senses if the handheld device is applied to the paper and the pressure between the handheld device and paper.
It is possible to transfer the information from the handheld device to an external computer in the nature of a video signal from the sensor and all computing taking place in the external computer. Alternatively, the handheld device may be provided with means for image processing. Thus, the handheld device may be provided with a processor and/or logic arrays, which processes the video signal obtained from the sensor and calculates the absolute position from the dots or markings imaged from the surface of the paper. This image processing comprises adjustments for perspective effects from rotation and inclination of the handheld device in relation to the paper and compensation for different light conditions. The handheld device may further comprise computer programs which calculates the x-y-coordinates of the handheld device on an imaginary surface.
Information may be stored in the memory of the handheld device in the nature of a coordinate train, which may be converted to a vector format. This message information may be transferred to a computer, which draws a graph on the screen of the computer by means of a computer program. The message information may be sent to a local or distant printer for direct printout or to a dumb display device which only has the ability to draw the graph at the screen, such as a TV screen.
Interpretation of the message information to a character format may take place in the handheld device or in a computer system.
In addition to coordinate information there is also time information, such as time information for each coordinate, which is recorded each hundredth of a second when the pressure sensor is activated. Moreover, there may be information about the pressure towards the support and inclination and rotation information for the position of the handheld device in relation to the support.
Message information may be communicated to a recepient in the nature of a fax or an e-mail. In this situation, the handheld device may cooperate with a mobile telephone via a wire, IR or radio link. The mobile telephone may operate as a “modem” for transferring the message information to a telefax apparatus at a telefax number, or to Internet via calling a modem pool of an Internet operator. In this case, it may be required that certain portions of the message information is character recognized, which may take place in the handheld device. Such recognized information may be used as a telefax number or an IP address for communication via Internet. Address information in the mobile telephone may alternatively be used.
The computer may perform the same functions as described above as performed in the handheld device.
The information may be transferred to a server via Internet at a preprogrammed IP address and the character recognition may take place in said server.
In the above situations, it may be desired to use the input device, i.e. the handheld device for selecting certain text and indicate some actions to be performed on such information. Actions may be qualifying actions, such as qualifiying certain properties of the selected information, such as color or line thickness etc. Actions may also be operations to be performed on the selected information, such as character recognition or encryption. Other types of actions may be to expose the information for a program for encapsulating the information to a e-mail standard format to be transmitted as an e-mail. Further types of actions may be to compile the selected information in a binder or folder in a computer file system, as further described below.
Thus, there is a need for a method and a system for associating information with other information, actions or qualifications.